Add a re-timing harness for the SLOW-vs-stall audit - #110
Merged
Conversation
A single run cannot tell a slow script from a stalling one: both miss the cap. A slow script has a tight timing distribution; a stalling one is bimodal, completing in tens of seconds when it completes at all. Telling them apart needs the same script run N times under the real cap, and nothing here could do that -- run_smoke.py runs smoke_tests.txt once. retime.py imports run_one and the env-profile resolution from run_smoke.py rather than reimplementing them, so this harness, the PR gate and the release runner cannot disagree about a script's environment or its cap. retime.yml is a workflow_dispatch caller into PyAutoHeart's reusable smoke-tests.yml via its new `runner` input, so the chain checkout and install epilogue are not duplicated either. Timeouts exit 0: here a timeout is the measurement, not a failure. For autolens_workspace_test#271. Requires PyAutoHeart#161.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For autolens_workspace_test#271 (phase 2 of the
jax-compile-stallepic). Blocked on PyAutoHeart#161.Why
autolens_workspace_test#271 has to establish whether 21 SLOW-marked JAX entries across this repo and
autolens_workspace_testare genuinely slow, or are the autolens_workspace_test#245 XLA compile stall wearing a different label. A single run cannot tell those apart — both miss the cap. The discriminator is the distribution:Nothing here could measure that.
run_smoke.pyrunssmoke_tests.txtonce, andsmoke_tests.ymlis apull_request/pushcaller — there is no way to run a chosen script N times.This repo carries three of the five stall quarantines:
multi_dataset/jax_likelihood/rectangular.py(NEEDS_FIX 2026-08-01),imaging/jax_likelihood/mge_group.pyandimaging/jax_likelihood/rectangular_mge.py(both NEEDS_FIX 2026-08-23), plus nine SLOW-marked JAX entries.What this adds
.github/scripts/retime.py— runs each named script N times, records every outcome, and reports a verdict per script (STALL/SLOW/NEITHER/AMBIGUOUS/ERROR) with the numbers behind it. Writes a job summary table andretime_results.json.It imports
run_oneand the env-profile resolution fromrun_smoke.pyrather than reimplementing them. That is the point:run_smoke.py's own docstring records what a copy cost last time, when a duplicatedload_env_confighardcoded the smoke profile path and left the PR gate unable to read the release profile. Onerun_onemeans this harness, the PR gate and the release runner cannot disagree about a script's environment or its cap..github/workflows/retime.yml— aworkflow_dispatchcaller takingscripts,repeatsandscript-timeout(300 = smoke, 1800 = release). Likesmoke_tests.ymlit is thin: the chain checkout, install epilogue and cache dirs stay in PyAutoHeart's reusable workflow, reached through the newrunnerinput from PyAutoHeart#161. The matrix runs both 3.12 and 3.13 by default, which matters here —imaging/jax_likelihood/rectangular_mge.pypassed on 3.13 and stalled on 3.12 for the identical commit, which is what put it inno_run.yaml.Timeouts exit 0. A timeout is this harness's measurement, not its failure. Exiting non-zero on one would make a red run the expected outcome and hide the runs that genuinely broke.
The classification line
BIMODAL_RATIO = 0.5— a median completion below half the cap is treated as a difference in kind rather than degree.imaging/jax_likelihood/mge_group.py's own marker records the siblingmge.pypassing in 9.4s against a 300s cap, which is 3%; a genuinely marginal script sits near 90%. It is a judgement, so the ratio that drove each verdict is always printed next to it and can be re-read from the JSON.Verification
classify()was exercised against the timings the existing markers already record, and returns the verdict the marker text implies in each case:multi_dataset/jax_likelihood/rectangular.py— "hung to the cap, passes ~19s otherwise"STALL(median 1% of cap)interferometer/datacube/shared_preloads.py(al_test) — 27.8s measured, "flakes at the 1800s cap"STALL(median 2% of cap)SLOW(median 95% of cap)NEITHERAMBIGUOUS— refuses to guess without a completion timeERRORThat is the harness reproducing conclusions we can already justify from the record; the real exercise is the first dispatch.
Order
PyAutoHeart#161 adds the
runnerinput this calls. A caller cannot pass an input that does not exist, so #161 merges first — until it does, a dispatch of this workflow will fail at workflow resolution. The sibling autolens_workspace_test#272 carries the identical harness with that repo's dependency chain.Generated by Claude Code